United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


10/663,236 


09/15/2003 


Eric J. Larsen 


SONYP029 


3753 



25920 7590 07/11/2008 

MARTINE PENILLA & GENCARELLA, LLP 
710 LAKEWAY DRIVE 
SUITE 200 

SUNNYVALE, CA 94085 



EXAMINER 



OMOTOSHO, EMMANUEL 



PAPER NUMBER 



3714 



07/11/2008 



DELIVERY MODE 



PAPER 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 



PTOL-90A (Rev. 04/07) 



\JI 1 /(/C MlrllUII OUflfflfCffy 


Application No. 

10/663,236 


Applicant(s) 

LARSEN ET AL. 


Examiner 

EMMANUEL OMOTOSHO 


Art Unit 

3714 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )^l Responsive to communication(s) filed on 28 April 2008 . 
2a )□ This action is FINAL. 2b)^ This action is non-final. 

3) \Z\ Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) ^3 Claim(s) 1,4-19,21 and 46-50 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) Q Claim(s) is/are allowed. 

6) E3 Claim(s) 1.4-19.21.46-50 is/are rejected. 

7) Q Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) L~H The specification is objected to by the Examiner. 

10) O The drawing(s) filed on is/are: a)0 accepted or b)0 objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) 0 The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) 0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)Q Some * c)Q None of: 

1 0 Certified copies of the priority documents have been received. 

20 Certified copies of the priority documents have been received in Application No. . 

3.D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attach ment(s) 

1 ) □ Notice of References Cited (PTO-892) 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) O Information Disclosure Statement(s) (PTO/SB/08) 

Paper No(s)/Mail Date . 



4) l~l Interview Summary (PTO-41 3) 

Paper No(s)/Mail Date. . 

5) O Notice of Informal Patent Application 

6) □ Other: . 



U.S. Patent and Trademark Office 

PTOL-326 (Rev. 08-06) 



Office Action Summary 



Part of Paper No./Mail Date 20080702 



Application/Control Number: 10/663,236 
Art Unit: 3714 



Page 2 



DETAILED ACTION 

Request for Continued Examination (RCE) 

This is in response to the RCE filed 4/28/08 in which claims 1,4, 13-14 and 46 
were amended. Claims 1, 4-19, 21 and 46-50 are pending. 

Claim Rejections - 35 USC § 103 

1 . The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

2. The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1 , 148 
USPQ 459 (1966), that are applied for establishing a background for determining 
obviousness under 35 U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating 
obviousness or nonobviousness. 

3. Claims 1-26,28-44 and 46-57 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kang US Patent 6,009,210 and further in view of Kanade et al. US 
Patent 6,1 51 ,009. 

4. Kang teaches a method for processing interactive user control for a view of a 
scene displayed on a virtual window comprising: 

5. In regards to claims 1,14,22,25,28,34,35,39,43,46,47,52,53,57 
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a. Identifying a head of a user that is to interact with the scene (Col 1 lines 
43-47 and Col 3 lines 29-32) 

b. Storing an initial frame of image data representing the head of the user 
(Col 3 lines 29-35) 

c. Tracking the identified head of the user during display of the scene, the 
tracking enabling detection of a change in position of the head of the user 
(Abstract) 

d. Tracking including identifying a search region within a frame of the user 
image data and comparing values within the search region to template values of 
the initial frame of the stored image data; adjusting a view frustum (which is 
being interpreted as the "view of the user into the virtual world") in accordance 
with the change in position of the head of the user (Col 2 lines 34-43 and Col 3 
lines 4-21) 

e. Repeating the identifying the search region, the comparing, and adjusting 
for successive frames of the scene, wherein the comparing is performed with the 
initial frame of the stored image data (Col 2 lines 34-43 and Col 9 lines 17-44) 

f. A computing device and a display screen in communication with the 
computing device configured to display image data defined through a view- 
frustum (Fig 1) 

g. Tracking device is a camera (Fig 1) 

6. In regards to claims 2,15,53, a view frustum initially defined by a triangular gaze 
projection set between outer edges of a virtual window and a virtual position of the head 
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when the virtual position of the head is normal to a center point of the virtual window 
(Figs 1-3) 

7. In regards to claim 3,16,23-24,53 adjusting the view-frustum moves the virtual 
position of the head away from normal relative to the center point of the virtual window 
(Col 2 lines 34-43) 

8. In regards to claim 4,17, virtual position of the head being away from normal 
relative to the center point of the virtual window changes an angle of the triangular gaze 
projection, the change in angle of the triangular gaze projection displays a change in 
viewing angle of the scene provided by the video clip (Col 4 line 60- Col 5 line 49) 

9. In regards to claim 5, the change in viewing angle of the scene is a result of the 
detected movement of the head of the user to enable the interaction with the scene (Col 
4 line 60- Col 5 line 49) 

10. In regards to claim 1,6-7,14,22-23,26,28-29,31,34,39,41,42,46,49,50,52 Kang fail 
to specifically teach 

h. Adjusting a scale of the scene according to a change in a distance of the 
head of the user from a capture device 

i. Using a capture device having depth-capturing capability. 

1 1 . Kanade teaches the use of a depth capturing camera for interaction with a view 
of a scene (abstract) 

12. One of ordinary skill in the art, at the time of the invention, would have been 
motivated to have a system that interacts with a view of a scene using depth capturing 
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device to adjust a scale of the scene according to a change in a distance of the head of 
the user from the capture device. Using a depth camera, the system would be able to 

j. Obtain distance or depth information from the object being tracked 

(inherent of "depth camera) 

k. Determine any interaction such as occluding, shadowing, reflecting or 
colliding (Kanade abstract) 

I. Generate appropriate output based on said determination (Kanade 
abstract) 

13. In regards to newly added limitations that further defined and claimed the 
definition of the view frustum defined by a triangular gaze projection set between 
outer edges of the virtual window and a position of the head when the position of 
the head is substantially normal to about a center point of the virtual window. 
Applicant should respectfully note that this is the known view projection 
associated with depth cameras. Depth cameras acquire and process recieved 
image data by performing a triangular coordinate transformation on each 
recieved pixel. The depth information is processed by reading the objects 
current location in reference to the windows center point (i.e. the need for 
calibration and storing initial image) and transforming the received location to an 
(x,y,z) coordinates. Where x and y corresponds to a position that relates to a 
position on the display screen and z corresponds to the distance from the 
position to camera. Hence the triangular projection. A constant focus on the 
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object has to be maintained throughout the process for the system to be able to 
maintain the correct coordinates. 

14. In regards to claim 8,30, the initial frame of image data is marker-less (Fig 1) 

1 5. In regards to claim 9, the initial frame of data is maintained throughout the video 
clip (Fig 1) 

16. In regards to claim 10,48,54 the video clip is of a video game. Although Kang 
teaches the method to be generally associated with virtual environment in computer 
systems, Kang further teaches that video games are well known in the art as virtual 
environment in a computer system (Col 1 lines 26-40) 

1 7. In regards to claim 1 1 , the interaction with the scene by tracking movement of the 
head of the user is independent of user hand-held controls for interacting with the video 
game (Col 1 lines 5-9) 

18. In regards to claim 1 2,32,40,55, the method operation of tracking the identified 
head of the user during display of the video clip includes tracking a facial portion of the 
head and matching gray scale image data associated with the facial portion to image 
associated with a template of the facial portion (Col 9 lines 20-33) 

1 9. In regards to claim 1 3,1 8-20,24,26,33,35,41 ,42,44,50,56, the method of adjusting 
a view frustum in accordance with the change in position of the head of the user 
includes identifying a point of interest of the scene of the video clip and modifying the 
view-frustum so that the point of interest appears at a constant position when displayed 
in successive video clips (Kang Claim 6). The Examiner should point out that each 
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frame in Kang's disclosure has to be scanned before the scene transformation could 
occur. 

20. In regards to claim 21 ,36-38,49-51 , the method operation of translating a view 
frustum in accordance with the change in position of the head of the user includes 
rotating the view-frustum about the head of a user according to the change in position of 
the head of the user (Kang Claim 5). 

Response to Arguments 

21 . Applicant's arguments filed 7/1 6/07 with respect to claims 1 -26,28-44 and 46-57 
have been considered but are moot in view of the new ground(s) of rejection. 

Response to Arguments 

22. Applicant's arguments filed 12/04/07 have been fully considered but they are not 
persuasive. 

23. On page 20, applicant argues, "Further, Kang would not adjust a scale of the 
scene according to a change in a distance of the head of the user from a capture device 
as specified in claims I, 14, 22, 28, 33, 39, 46 and 52, or include a camera having depth 
capability as specified in claims 7 and 31, since the change in relative object depth 
compared to the distance to the camera must be small under the affine model (see 
column 5 lines 58-63). For the affine model to work, it is required that the surface 
appear planar. In order for the face to appear planar, the distance from the user to the 
camera must be relatively large. The use of a depth camera would render moot the 
entire affine model principles for which Kang is constructed" 



Application/Control Number: 10/663,236 Page 8 

Art Unit: 3714 

24. The examiner respectfully disagrees. Kang's invention is directed to 2-D affine 
"transformation". Kang teaches that the technique is suitable for planar surfaces with 
an affine camera model especially when the user is far from the camera. Therefore, a 
depth camera would not render moot Kang's affine model technique. Please see Col 
6:44-60 of Kang. 

25. On page 21 , applicant argues, "The Examiner indicates that the teachings of 
Kanade would motivate one skilled in the art to interact with a view of a scene using 
depth capturing to adjust a scale of the scene according to a change in distance of the 
head of the user. Although Kanade teaches the identification of depth, the depth is only 
needed to capture the user, who needs to stand in from of a backdrop. These teachings 
do not teach to adjust the scale of the scene. In fact, the position of the user's head is in 
no way tied to any adjustment in a scene. The only distance tracking done is to identify 
where the user is located, so that objects can be positioned in and around the scene. 
As a further distinction, the "scene", as claimed is different from the area captured by 
Kanade. Specifically, in the claimed invention, the view-frustum is adjusted in 
accordance with the change in position of the head of the user and the adjusting of a 
scale of the scene takes placed according to a change in a distance of the head of the 
user from a capture device. In Kanade, the user is "in the scene". That is, the user is 
captured, and then injected into the scene. What the user does in the scene in no way 
performs adjustments. Indeed, if the user moves his head, the scene is not adjusted." 

26. However, Kanade is not being relied upon for the teaching of adjusting the scale 
of the scene. As shown above, Kang teaches adjusting the scale of the scene based on 
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user location information received from a camera. Kanade is being used to show that 
using depth capturing image to capture depth information of a user during a gaming 
session is old in the art (Kanade abstract). 

Response to Arguments 
27. Applicant's amendments/arguments filed 4/28/08 have been fully considered. 
Please see paragraph 13 above which addresses the newly added limitations. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to EMMANUEL OMOTOSHO whose telephone number is 
(571)272-3106. The examiner can normally be reached on m-f 10-6. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Robert Pezzuto can be reached on (571) 272-6996. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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